Cloud PubSub

Cloud PubSub is a managed processing queue/messaging service which works on the Publisher and Subscriber model. A publisher publishes the message to a particular topic and all the subscribers which are subscribed to that topic receive the message until the acknowledgment is not sent.

PubSub guarantees that message will be sent to the subscribers at least once. You can consider PubSub as an alternative to Apache Kafka which is an open-source alternative.

Let’s explore more about Cloud PubSub.

Introduction#

The PubSub is used to decouple services from each other and enable them to scale independently. The services can communicate with each other using the queue and send updates to all other services which are dependent on them.

Apart from this, PubSub can also be used to gather real-time data from different IoT devices and push it to BigQuery for analysis. The Cloud PubSub is integrated with most of the GCP services by default. You can easily access the integration on the home page of the service itself. But for now, we will focus on Cloud PubSub only.

Let’s understand the PubSub workflow in detail.

Working#

Cloud PubSub architecture has 6 components.

1. Topic: Logical queue which acts as an interface between subscribers and publishers. The topic is a way to logically organize the communication channel between subscribers and publishers.

   For example, We can have a topic named “Music” where all the music-related messages are published. If any subscriber wants to get updates related to the music only, then it can subscribe to that channel/topic.

2. Publisher: In the pub-sub architecture, the first part pub stands for the publisher. The Publisher is something that sends out messages to the queue. It can be any HTTP client that can make HTTP requests to send messages to the queue.

3. Subscription: A named resource representing the stream of messages from a single, specific topic, to be delivered to the subscribing application. A subscriber (can be any application that can use the messages to perform some action.) subscribes to the subscription to receive messages.

   The subscription is the only interface to consume messages. There will be only one topic for each subscription.

4. Message: The combination of data and (optional) attributes that a publisher sends to a topic and is eventually delivered to subscribers.

5. Message attribute: A key-value pair that a publisher can define for a message. You can also consider this as metadata for messages. For example, key language_tag and value en could be added to messages to mark them as readable by an English-speaking subscriber.

6. Subscriber: The subscriber is any application that subscribes to the topic using subscription. A subscriber can use more than one subscription to receive messages from more than one topic. Also, there can be more than one subscriber-specific to each topic as well. The implementation depends on the business logic.

Flow of message#

The typical flow of the message is from,

Publisher > Topic > Message Storage > Subscription > Subscriber.

If the Acknowledgement is not sent for the message by the subscriber then the Cloud Pubsub delivers the message one more time and save the message in the message store until the expiration period. This will increase the unacknowledged message count and new messages start piling up. Hence sending the acknowledgement for messages is recommended.

The message delivery is totally dependant on the type of subscription used. The Cloud PubSub provides 2 types of message deliveries based on the subscription types.

1. Pull Subscription: When the pull method of message delivery is used, you need to create a pull subscription, the subscriber needs to requests upcoming messages using that subscription. Either subscriber can continuously poll for incoming messages or use periodic pulls.

2. Push: When the push method of message delivery is used, you need to specify one publicly accessible HTTPS endpoint. The messages are delivered to that endpoint using an HTTP request. The delivery of messages is immediate.

You can also configure the retention of unacknowledged messages and a subscription deleting policy to automatically delete inactive subscriptions.

Dashboard#

Let’s look at the Cloud PubSub dashboard as per the steps mentioned in the Flow of message section.

To access the Pub/Sub service in Google Cloud, open the main navigation menu (top left hamburger icon) > MORE PRODUCTS > ANALYTICS > Pub/Sub. There will be no topic hence you will see an empty dashboard with buttons to create the topic.

1. Click on the Create Topic button. This will open up a form to create the topic. Name the topic. By default, the Add a default subscription checkbox will be checked. Uncheck the box as we will create the subscription later on.

2. As soon as the topic is created, you will be redirected to the topic dashboard. You can see different sections here. But focus on the PUBLISH MESSAGE button. Click the publish message button to open the form to publish the message.

3. The form will have the following input fields:

• Number of messages: How many times the message should be delivered. Keep it 1.

• Message interval: If the Number of messages field has a number more than 1, then we need to define after how many intervals the message should be delivered again. The field will be disabled because we will be delivering only once.

• Message body: You can use this text box to send any message. Message can be plain text or JSON. Pubsub delivers all messages as a byte string.

  There will be other extra fields like Message attributes and Message ordering but these are not required as of now and the use of these fields comes under the Cloud Architect course.

  Once you write the message click on the Publish button.

Since we don’t have any subscription created, we won’t be able to consume or view this published message. This message will be deleted as no subscription is registered for this topic. So let’s create the subscription so that whenever we publish a message the message will be stored until a subscriber pulls the message.

4. Go back to the topic dashboard. If you are following as it is then the topic name will be gcp-course-demo. Click on the VIEW MESSAGES button. As we have seen in the introduction section, we need a subscription to view the message. Click on the CREATE A SUBSCRIPTION button.

5. Name the subscription. The standard way is to name the subscription using the [Topic-name]-subscription format. You can name it anything. But following standards is recommended. Next is to select the delivery type for the subscription. Select the Pull type. The rest of the fields can be kept default. Scroll down and click the CREATE button. After the subscription is created, you will be redirected to the subscription dashboard.

6. If you click on the VIEW MESSAGES button on the subscription dashboard, an interface to pull the messages will open up. You won’t be able to see/pull the previously published message because Cloud Pubsub only sends the messages which are published after the subscription is created.

7. Now that we have our end-to-end message flow control components ready, let’s publish the message one more time and try to view it using the created subscription.

• In the left side pane click Topics > [Topic name].

• Click the Publish Message button and repeat step 3 or write the message you want to publish. Once you publish the message by clicking the Publish button, you will be redirected back to the topic dashboard.

• Click on the VIEW MESSAGES button. Select the subscription from the subscription list.

• Check the Enable ack messages check box so that a button to acknowledge the message will be displayed for 10 seconds(This is the acknowledgment deadline).

• Click on the PULL button. You will see the published message.

This is the basic demo of the PubSub but it is used for more than this. As it is listed under the BigData, Pubsub is also an essential service for processing streaming real-time data. Following are the different use cases of PubSub.

Usecases#

• Stream Analytics: Google’s stream analytics makes data more organized, useful, and accessible from the instant it’s generated. Cloud PubSub is used to gather continuous streaming data from mobiles, IoT devices, laptops, and other network devices for this pipeline.

• Asynchronous microservices integration: Pub/Sub works as a messaging middleware for traditional service integration or a simple communication medium for modern microservices. By looking at the topics dashboard you can imagine the integration of the PubSub with other services.

For the exam, whenever you come across words like Capture Streaming data, Pubsub, Decoupled and Asynchronous application architecture then Cloud PubSub is the answer.

Lab#

You have learned the basics of PubSub. Complete this interesting lab to strengthen your understanding. You might not want to skip this.

• Empower Your Gmail Inbox with Google Cloud Functions and Cloud PubSub

Make sure you delete all the resources after the lab to avoid getting charged.